Wemodel 34], whichisformallyG = hS, U, Pi. S is thestatespacet, every i 2 A {1,..., n} choosesui 2 U which action u 2 U Un. P(s0|s,u): S U S!

InProceedingsofthe First International Conferenceon Distributed Artificial Intelligence, pages 1-7, 2019.

Exploration in multi-agent reinforcement learning is a challenging problem, especially inenvironments with sparse rewards.

In this paradigm of centralised training for decentralised execution, QMIX [25] is a popular Qlearning algorithm with state-of-the-art performance ontheStarCraft Multi-Agent Challenge [26]. QMIX represents the optimal joint action value function using a monotonicmixing function of per-agent utilities.

This is challenging from an implementation perspective, as repeatedly differentiating policy gradient estimates may lead to biased Hessian estimates.